Image resolution increasing method and apparatus

ABSTRACT

An image resolution increasing method include setting a first block which is included in a low resolution image and is located at a first position, and a second block which is included in a high resolution image and is located at a second position, and setting, as an increasing resolution block of the first block, a third block expressed by a second vector obtained by projecting a first vector representing the second block to a linear manifold as a set of vectors that indicate fourth blocks of the second block size, the fourth blocks becoming the first block due to reduced resolution, in a Euclidean space having, as the number of dimensions, a product of the number of pixels arranged vertically in the second block size and the number of pixels arranged horizontally in the second block size.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromprior Japanese Patent Application No. 2007-333301, filed Dec. 25, 2007,the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image resolution increasing methodand apparatus, which convert image data captured by a camera or receivedby a television into image data of a higher resolution.

2. Description of the Related Art

In recent years, display apparatuses such as televisions, displays, andthe like having a large number of pixels have prevailed. An image can bedisplayed up to its details as the number of pixels of the displayapparatus is larger. That is, a high-resolution image can be displayed.

In order to increase the number of pixels, an interpolation manner ispopularly used. The interpolation manner can speed up processing but itcannot always obtain a sharp image. In order to obtain a sharper imagethan the interpolation manner, a method of generating a high-resolutionimage that satisfies a constraint called a reconstruction constraint isavailable (for example, see A. Tanaka, H. Imai, and M. Miyakoshi,“Digital Image Enlargement Based on Kernel Component Estimation,”International Journal of Computing Anticipatory Systems, vol. 15, pp.97-108, 2004).

However, the conventional method of generating a high-resolution imagethat satisfies a reconstruction constraint suffers a problem of highcalculation cost. In case of high calculation cost, for example, in asituation that requires increasing resolution so as to display a movingpicture on a display apparatus, the processing speed of increasingresolution becomes lower than the playback speed of the moving picture,thus posing a problem of dropping frames.

BRIEF SUMMARY OF THE INVENTION

In accordance with a first aspect of the invention, there is provided animage resolution increasing method, comprising: inputting alow-resolution image having a first size as a size that indicates thenumbers of pixels arranged vertically and horizontally; inputting ahigh-resolution image obtained by increasing in resolution thelow-resolution image to a second size by an interpolation manner or asuper-resolution manner, the second size being a desired size of a firstimage finally generated; calculating a first block size corresponding tothe first size and a second block size corresponding to the second sizein accordance with the first size and the second size; setting a firstposition of a block of the first block size in the low-resolution image,and a second position of a block of the second block size in thehigh-resolution image or the first image, the second positioncorresponding to the first position; setting a first block which isincluded in the low-resolution image and is located at the firstposition, and a second block which is included in the high-resolutionimage and is located at the second position; setting, as an increasedresolution block of the first block, a third block expressed by a secondvector obtained by projecting a first vector representing the secondblock to a linear manifold as a set of vectors that indicate fourthblocks of the second block size, the fourth blocks becoming the firstblock due to reduced resolution, in a Euclidean space having, as thenumber of dimensions, a product of the number of pixels arrangedvertically in the second block size and the number of pixels arrangedhorizontally in the second block size to obtain a plurality of increasedresolution blocks; and generating the first image using at least one ofthe increased resolution blocks set by changing the first position andthe second position.

In accordance with a second aspect of the invention, there is providedan image resolution increasing method, comprising: inputting alow-resolution image having a first size as a size that indicates thenumbers of pixels arranged vertically and horizontally; inputting ahigh-resolution image obtained by increasing in resolution thelow-resolution image to a second size by an interpolation manner or asuper-resolution manner, the second size being a desired size of a firstimage finally generated; calculating a first block size corresponding tothe first size and a second block size corresponding to the second sizein accordance with the first size and the second size; setting a firstposition of a block of the first block size in the low-resolution image,and a second position corresponding to the first position of the secondblock size in the high-resolution image or the first image; setting afirst block which is included in the low-resolution image and is locatedat the first position, and a second block which is included in thehigh-resolution image and is located at the second position; setting, asan increased resolution block of the first block, a third blockexpressed by a vector corresponding to points on a straight line thatpasses through two points indicated by a first vector and a secondvector, the first vector representing the second block, the secondvector being obtained by projecting the first vector to a linearmanifold as a set of vectors that indicate fourth blocks of the secondblock size, the fourth blocks becoming the first block due to reducedresolution, in a Euclidean space having, as the number of dimensions, aproduct of the number of pixels arranged vertically in the second blocksize and the number of pixels arranged horizontally in the second blocksize to obtain a plurality of increased resolution blocks; andgenerating the first image using at least one of the increasedresolution blocks set by changing the first position and the secondposition.

In accordance with a third aspect of the invention, there is provided animage resolution increasing method, comprising: calculating a firstblock size corresponding to a first size and a second block sizecorresponding to a second size in accordance with the first size and thesecond size, the first size being a size of a low-resolution image and asize that indicates the numbers of pixels arranged vertically andhorizontally, the second size being a desired size of a first imagefinally generated; setting a first position of a block of the firstblock size in the low-resolution image, and a second position of a blockof the second block size in the first image, the second positioncorresponding to the first position; setting a first block which isincluded in the low-resolution image and is located at the firstposition; calculating, using the first block size and the second blocksize, a first coefficient used to generate a second block of the secondblock size by an interpolation manner from the first block; calculating,using the first coefficient, the first block size, and the second blocksize, a second coefficient used to calculate a second vector obtained byprojecting a first vector indicating the second block to a linearmanifold as a set of vectors that indicate third blocks of the secondblock size, the third blocks becoming a part of the first block due toreduced resolution, in a Euclidean space having, as the number ofdimensions, a product of the number of pixels arranged vertically in thesecond block size and the number of pixels arranged horizontally in thesecond block size; calculating the second vector corresponding to aposition indicated by the second position from the first block and thesecond coefficient, setting a fourth block indicated by the secondvector to an increased resolution block as a part of the first block toobtain a plurality of increased resolution blocks; and generating thefirst image using at least one of the increased resolution blocks set bychanging the first position and the second position.

In accordance with a fourth aspect of the invention, there is providedan image resolution increasing method, comprising: calculating a firstblock size corresponding to a first size and a second block sizecorresponding to a second size in accordance with the first size and thesecond size, the first size being a size of a low-resolution image and asize that indicates the numbers of pixels arranged vertically andhorizontally, the second size being a desired size of a first imagefinally generated; setting a first position of a block of the firstblock size in the low-resolution image, and a second position of a blockof the second block size in the first image, the second positioncorresponding to the first position; setting a first block which isincluded in the low-resolution image and is located at the firstposition; calculating, using the first block size and the second blocksize, a first coefficient used to generate a second block of the secondblock size by an interpolation manner from the first block; calculating,using the first coefficient, the first block size, and the second blocksize, a second coefficient used to calculate a second vector obtained byprojecting a first vector indicating the second block to a linearmanifold as a set of vectors that indicate third blocks of the secondblock size, the third blocks becoming a part of the first block due toreduced resolution, in a Euclidean space having, as the number ofdimensions, a product of the number of pixels arranged vertically in thesecond block size and the number of pixels arranged horizontally in thesecond block size; calculating the second vector corresponding to aposition indicated by the second position from the first block and thesecond coefficient; setting, as an increased resolution block as a partof the first block, a fourth block expressed by a vector correspondingto points on a straight line that passes through two points indicated bythe first vector and the second vector to obtain a plurality ofincreased resolution blocks; and generating the first image using atleast one of the increased resolution blocks set by changing the firstposition and the second position.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a block diagram of an image resolution increasing apparatusaccording to the first embodiment;

FIG. 2 is a flowchart showing an example of the operation of the imageresolution increasing apparatus shown in FIG. 1;

FIG. 3 is a view showing a state in which a high-resolution image isreduced in resolution to a low-resolution image;

FIG. 4 is a view for explaining the geometric significance of the imageresolution increasing method according to the first embodiment;

FIG. 5 is a block diagram of an image resolution increasing apparatusaccording to the second modification of the first embodiment;

FIG. 6 is a flowchart showing an example of the operation of the imageresolution increasing apparatus shown in FIG. 5;

FIG. 7 is a view showing an example of a state in which ahigh-resolution image is reduced in resolution to a low-resolution imageaccording to the sixth modification of the first embodiment;

FIG. 8 is a view for explaining the positional relationship betweenlow-resolution blocks in a low-resolution image;

FIG. 9 is a view for explaining the positional relationship betweenhigh-resolution blocks in a high-resolution image;

FIG. 10 is a view for explaining the positional relationship betweenhigh-resolution blocks in a high-resolution image upon application ofthe sixth modification of the first embodiment;

FIG. 11 is a block diagram of an image resolution increasing apparatusaccording to the second embodiment;

FIG. 12 is a flowchart showing an example of the operation of the imageresolution increasing apparatus shown in FIG. 11;

FIG. 13 is a view for explaining the relationship between y_(p) andz_(p);

FIG. 14 is a view for explaining the relationship between y_(p) andx_(p);

FIG. 15 is a view for explaining a state in which resolution increasingis independently applied vertically and horizontally by the imageresolution increasing method according to the second embodiment;

FIG. 16 is a block diagram of an image resolution increasing apparatusaccording to the second modification of the second embodiment;

FIG. 17 is a flowchart showing an example of the operation of the imageresolution increasing apparatus shown in FIG. 16;

FIG. 18 is a block diagram of a coefficient calculation apparatusincluded in an image resolution increasing apparatus according to thesixth modification of the second embodiment; and

FIG. 19 is a block diagram of a resolution increasing calculationapparatus included in the image resolution increasing apparatusaccording to the sixth modification of the second embodiment.

DETAILED DESCRIPTION OF THE INVENTION

An image resolution increasing method and apparatus according toembodiments will be described in detail hereinafter with reference tothe accompanying drawing. Note that components denoted by the samereference numbers perform the same operations, and a repetitivedescription thereof will be avoided. The embodiments are not limited tothose described below, but they may be changed. For example, theembodiments may be changed based on modifications described later or bycombining these modifications.

Terms will be explained first.

A digital image is defined by values each having information of a coloror luminance at respective positions. A unit having this value is calleda pixel, and its value is called a pixel value. For the sake ofsimplicity, a case will be explained below wherein a pixel has only avalue representing luminance as a pixel value. In this case, one pixelhas one pixel value. When a pixel has values each having colorinformation as pixel values, for example, the pixel may have values, aspixel values, on an RGB color space that expresses information of colorsassociated with red, green, and blue on a three-dimensional space. Inthis case, one pixel has three values associated with red, green, andblue, as pixel values.

In order to increase the resolution of a digital image, the number ofpixels needs to be increased. The processing for increasing the numberof pixels is called resolution increasing of an image in some cases.Also, enlargement and resolution increasing of an image may be used asthe same meaning, and size degradation and reduced resolution of animage may be used as the same meaning.

A rectangular partial image in an image is called a block. In thefollowing description, a block is expressed by a vector defined byarranging pixel values in that block. For the sake of simplicity, avector may be called a block and vice versa without distinguishing theblock and vector from each other.

FIRST EMBODIMENT

An image resolution increasing apparatus of this embodiment will bedescribed below with reference to FIG. 1.

An image resolution increasing apparatus 100 of this embodiment includesa low-resolution image input unit 101, high-resolution image input unit102, setting unit 103, initialization unit 104, and generation unit 105.The image resolution increasing apparatus 100 generates and outputs anincreased resolution image 153 based on image number data 156, size data160, a low-resolution image 151, and a high-resolution image 152. In thefollowing description, let w_(l) be the number of pixels (width)horizontally of a low-resolution image (still picture or moving picture)to be increased in resolution, and h_(l) be the number of pixels(height) arranged vertically. Also, let w_(h) be the width of an imageobtained by increasing in resolution the low-resolution image, and h_(h)be the height.

The low-resolution image input unit 101 inputs, as a low-resolutionimage 151, an n-th (n=1, 2, . . . , N) low-resolution image (movingpicture or still picture) of those to be increased in resolution. Thelow-resolution image 151 is data that represents pixel values ofrespective pixels. If a low-resolution image to be increased inresolution is encoded by, e.g., jpeg, mpeg2, or the like, thelow-resolution image input unit 101 comprises a codec for decoding theencoded image. The low-resolution image input unit 101 acquires blockdata 158 set by the setting unit 103, extracts a low-resolution block155 indicated by the block data 158 from the low-resolution image 151,and passes the low-resolution block 155 to the generation unit 105. Theblock data 158 is data representing the position of the low-resolutionblock 155 having a width c_(l) and height r_(l) in the low-resolutionimage 151. This position indicates, for example, the upper leftcoordinate position of a block. The setting unit 103 designates theblock data 158 while shifting its position. More specifically, thesetting unit 103 shifts the position by c_(l) horizontally and r_(l)vertically. In this way, the low-resolution blocks 155 in thelow-resolution image 151 are processed in turn.

The high-resolution image input unit 102 inputs the high-resolutionimage 152 obtained by increasing in resolution the low-resolution image151. The high-resolution image 152 is data that represents pixel valuesof respective pixels of an image having the same size as the increasedresolution image 153, which is generated and output by the imageresolution increasing apparatus 100, and an image obtained by increasingin resolution the n-th low-resolution image 151 is input as thehigh-resolution image 152. The resolution increasing method can beimplemented by existing interpolation manners, and for example, abilinear manner, cubic convolution manner, bicubic manner, cubic splinemanner, and the like allow high-speed processing. Also,reconstruction-based super-resolution, example learning-basedsuper-resolution, and the like may be used. If the high-resolution image152 is encoded by, e.g., jpeg, mpeg2, or the like, the high-resolutionimage input unit 102 comprises a codec for decoding the encoded image.The high-resolution image input unit 102 acquires block data 159 set bythe setting unit 103, extracts a high-resolution block 157 indicated bythe block data 159 from the high-resolution image 152, and passes thehigh-resolution block 157 to the generation unit 105. The block data 159is data which indicates the position of the high-resolution block 157having a width c_(h) and height r_(h) in the high-resolution image 152.This position is, for example, the upper left coordinate position of theblock. The setting unit 103 designates the block data 159 at a positioncorresponding to the block data 158. More specifically, the setting unit103 shifts the position by c_(h) horizontally and r_(h) vertically.

The initialization unit 104 acquires the size data 160, and outputsblock size data 154 and generation data 162. The size data 160 is datawhich represents the size (the number of pixels arranged horizontally,and that arranged vertically) of the low-resolution image 151, and thesize of the increased resolution image 153 to be generated. When alow-resolution image to be increased in resolution (moving picture orstill picture) is encoded by, e.g., jpeg, mpeg2, or the like, the sizedata 160 is a part of a header. The block size data 154 is calculatedfrom the input size data 160 (w_(l), h_(l), w_(h), and h_(h)). The blocksize data 154 is data which represents the size (width c_(l) and heightr_(l)) of a block in the low-resolution image 151, and the size (widthc_(h) and height r_(h)) of a block in the high-resolution image 152 orincreased resolution image 153 as processing units for resolutionincreasing. A method of calculating the block size data 154 (c_(l),r_(l), c_(h), and r_(h)) will be described later in (1-1). Thegeneration data 162 is data required for the generation unit 105, and iscalculated from the block size data (c_(l), r_(l), c_(h), and r_(h)).Details of the generation data 162 and its calculation method will bedescribed later in (1-2).

The setting unit 103 acquires the image number data 156, size data 160,and block size data 154, which is calculated by the initialization unit104. The image number data 156 is data which represents the number N oflow-resolution images to be increased in resolution. If a low-resolutionimage to be increased in resolution is a moving picture, and that movingpicture is encoded by, e.g., mpeg2 or the like, the image number data156 is a part of a header. The setting unit 103 acquires these data, andcalculates the block data 158, 159, and 161. The block data 161 is datawhich represents the position of an increased resolution block having awidth c_(h) and height r_(h) in the increased resolution image 153. Thesetting unit 103 designates the block data 161 at a positioncorresponding to the block data 158. More specifically, the setting unit103 shifts the position by c_(h) horizontally and r_(h) vertically. Byassociating the block data 158, 159, and 161 with each other, theprocessing can be done for each image block in place of the entireimage. The setting unit 103 checks if all increased resolution blocks inthe increased resolution image 153 have been processed. Furthermore, thesetting unit 103 checks based on the image number data 156 if all (Nimages; N frames) of low-resolution images (moving picture frames orstill pictures) to be increased in resolution have been processed.

The generation unit 105 generates an increased resolution block in theincreased resolution image 153, which is indicated by the block data161, based on the low-resolution block 155, high-resolution block 157,generation data 162, and block data 161. A method of generating theincreased resolution block will be described later in (1-3). Thegeneration unit 105 generates the increased resolution image 153 bygenerating all increased resolution blocks in the increased resolutionimage 153.

An example of the operation of the image resolution increasing apparatus100 shown in FIG. 1 will be described below with reference to FIG. 2.

(Step S201) The image number data 156 is input to the setting unit 103,and the size data 160 is input to the initialization unit 104 andsetting unit 103. The initialization unit 104 then calculates the blocksize data 154 and sends the calculated data to the setting unit 103.Finally, the initialization unit 104 calculates the generation data 162,and sends the calculated data to the generation unit 105.

(Step S202) The low-resolution image input unit 101 receives an n-thimage of low-resolution images (moving picture frames or still pictures)to be increased in resolution as the low-resolution image 151.

(Step S203) The high-resolution image input unit 102 receives thehigh-resolution image 152 obtained by increasing in resolution thelow-resolution image 151.

(Step S204) The setting unit 103 sends the block data 158 to thelow-resolution image input unit 101, the block data 159 to thehigh-resolution image input unit 102, and the block data 161 to thegeneration unit 105. The low-resolution image input unit 101 sends thelow-resolution block 155 in the low-resolution image 151, which isindicated by the block data 158, to the generation unit 105. Thehigh-resolution image input unit 102 sends the high-resolution block 157in the high-resolution image 152, which is indicated by the block data159, to the generation unit 105. The block data 158 is designated whileshifting its position upon repetitively processing step S204. Byassociating the block data 158, 159, and 161 with each other, theprocessing is done for each image block in place of the entire image.

(Step S205) The generation unit 105 generates an increased resolutionblock in the increased resolution image 153 indicated by the block data161, based on the low-resolution block 155, high-resolution block 157,and generation data 162.

(Step S206) The setting unit 103 checks if all increased resolutionblocks in the increased resolution image 153 have been processed. If allthe blocks have been processed, the process advances to step S207;otherwise, the process returns to step S204. Step S205 is repeated whilechanging the block data 158, 159, and 161 controlled by the setting unit103. As a result of repetition, the generation unit 105 generates andoutputs the increased resolution image 153 by generating all increasedresolution blocks in the increased resolution image 153. Note that endblocks are hard to receive attention, and are not often important.Hence, these blocks may be generated by a method requiring lowercalculation cost.

(Step S207) The setting unit 103 checks based on the image number data156 if all (N images) of low-resolution images (moving picture frames orstill pictures) to be increased in resolution have been processed. Ifall the images (frames) have been processed, the processing ends;otherwise, the process returns to step S202.

(1-1) Calculation Method of Block Size Data 154 (c_(l), r_(l), c_(h),and r_(h)) Initialization Unit 104

The initialization unit 104 calculates the height r_(l) of alow-resolution block and the height r_(h) of a high-resolution blockbased on a resolution increasing ratio vertically. The resolutionincreasing ratio vertically is h_(h)/h_(l). This fraction is reduced toan irreducible fraction. The denominator of this irreducible fraction isused as the height r_(l) of the low-resolution block, and the numeratoris used as the height r_(h) of the high-resolution block. In order toreduce the fraction to the irreducible fraction, for example, a greatestcommon factor of h_(h) and h_(l) is calculated by the Euclidean mutualdivision, and the numerator and denominator of h_(h)/h_(l) arerespectively divided by that greatest common factor. The initializationunit 104 calculates the width c_(l) of the low-resolution block, and thewidth c_(h) of the high-resolution block based on a resolutionincreasing ratio horizontally. The resolution increasing ratiohorizontally is w_(h)/w_(l). This fraction is reduced to an irreduciblefraction. The denominator of this irreducible fraction is used as thewidth c_(l) of the low-resolution block, and the numerator is used asthe width c_(h) of the high-resolution block. In order to reduce thefraction to the irreducible fraction, for example, a greatest commonfactor of w_(h) and w_(l) is calculated by the Euclidean mutualdivision, and the numerator and denominator of w_(h)/w_(l) arerespectively divided by that greatest common factor.

(1-2) Calculation Method of Generation Data 162 Initialization Unit 104

The process of reduced resolution of a high-resolution block x in ahigh-resolution image X to a low-resolution block y in a low-resolutionimage Y is described using a reduced resolution matrix W by:y=Wxwhere x is a (c_(h)r_(h))-dimensional column vector defined by arrangingpixel values of pixels of the high-resolution block, y is a(c_(l)r_(l))-dimensional column vector defined by arranging pixel valuesof pixels of the low-resolution block, and W is, for example, equal to amatrix that represents an area average determined by an area ratio ofcorresponding pixels. In an example of FIG. 3 (c_(l)=1, r_(l)=1,c_(h)=4, and r_(h)=4) in which a 4×4 high-resolution block is reduced inresolution to a 1×1 low-resolution block, the matrix that represents thearea average is a 1 (row)×16 (columns) matrix, and all elements of thismatrix are “ 1/16”s. The matrix that represents the area average neednot always be adopted as W. For example, a matrix that represents aweighted area average, in which weights become larger toward the centerof pixels may be adopted. Alternatively, a matrix in consideration ofoptical characteristics of a given camera may be adopted. Since theresolution of y is lower than that of x, c_(l)r_(l)<c_(h)r_(h) holds. Aresolution increasing problem is treated as a problem of reconstructingthe high-resolution block x which meets a reconstruction constraintgiven by y=Wx from the input low-resolution block y, and paving thathigh-resolution block x on the entire high-resolution image X.

The high-resolution block x is one point in a (c_(h)r_(h))-dimensionalEuclidean space expressed by:R^(c) ^(h) ^(r) ^(h) .That is, x is indicated by a (c_(h)r_(h))-dimensional vector in this(c_(h)r_(h))-dimensional Euclidean space. This (c_(h)r_(h))-dimensionalEuclidean space expressed by:R^(c) ^(h) ^(r) ^(h)is an orthogonal direct sum of R(W*) and N(W). That is, the equationbelow holds:R ^(c) ^(h) ^(r) ^(h) =R(W*)⊕+N(W)where W* is a conjugate transposed matrix of W. Since W is a real numbermatrix, W* is a transposed matrix. R(W*) is a range of W* defined by:R(W*)={x|x=W*y}.N(W) is a null-space of W defined by:N(W)={x|Wx=0}where “0” represents the (c_(l)r_(l))-dimensional column vectorincluding elements of all “0”s.⊕represents an orthogonal direct sum. From the relation of thisorthogonal direct sum, an arbitrary element of the Euclidean spaceexpressed by:R^(c) ^(h) ^(r) ^(h)can be uniquely decomposed into those of R(W*) and N(W). Therefore, thehigh-resolution block x can be decomposed into:x=P _(R(W*)) x+P _(N(W)) xwhereP_(R(W*))expresses an orthogonal projection matrix to R(W*), andP _(R(W*)) =W ⁺ Wholds. W⁺ is a Moore-Penrose generalized inverse matrix, and the matrixsize is (c_(h)r_(h)) rows×(c_(l)r_(l)) columns. P_(N(W)) represents anorthogonal projection matrix to N(W), andP _(N(W)) =I−W ⁺ Wholds. I is a unit matrix of (c_(h)r_(h)) rows×(c_(h)r_(h)) columns.Therefore, the high-resolution block x can be decomposed to:x=W ⁺ Wx+(I−W+W)x.x which satisfies the reconstruction constraint y=Wx can be decomposedto:x=W ⁺ y+(I−W ⁺ W)x.The first term of the right-hand side is an R(W*) component of x, andthe second term is an N(W) component of x. Therefore, a generalizedform, i.e., a generalized solution of the high-resolution block x whichsatisfies the reconstruction constraint y=Wx can be expressed by:x=W ⁺ y+(I−W ⁺ W)zwhere z is an arbitrary high-resolution block, and is a(c_(h)r_(h))-dimensional column vector. The R(W*) component of thisgeneralized solution is uniquely defined. On the other hand, since theN(W) component includes the arbitrary high-resolution block z, and isnot uniquely defined, the N(W) component of x is not uniquely defined.Therefore, there are a plurality of “x”s which satisfy thereconstruction constraint, and are not uniquely defined. FIG. 3 shows anexample in which high-resolution blocks x₁ and x₂ are reduced inresolution to an identical low-resolution block y by reduced resolutionexpressed by the reduced resolution matrix W. In this example, theresolution is reduced to 1/16 (¼ vertically and horizontally). As canalso been seen from FIG. 3, the high-resolution block x which satisfiesthe reconstruction constraint is not uniquely defined from thelow-resolution block y. Hence, in the embodiments, in order to acquire asharp high-resolution block with low calculation cost, z is calculatedfast from y by the interpolation manner, and a high-resolution blockx_(p) is calculated from z_(p) by:x _(p) =W ⁺ y+(I−W ⁺ W)z _(p).The first term of the right-hand side is an R(W*) component of x_(p),and the second term is an N(W) component of x_(p). The first term (R(W*)component) of this equation (x_(p)=W⁺y+(I−W⁺W)z_(p)) corresponds to acase in which a generalized inverse filter described in “HidemitsuOgawa, “Reconstruction of Signal and Image [III]-Projection Filters forOptimal Reconstruction-,” Journal of IEICE, vol. 71, pp. 739-748, 1988”is applied to resolution increasing of an image for respective blocks.FIG. 4 is a view for explaining the geometric significance of thecalculation of x_(p) from z_(p). In this example, since c_(h)r_(h)=3,R^(c) ^(h) ^(r) ^(h)is a three-dimensional Euclidean space. Each of z_(p) and x_(p) is onepoint in that three-dimensional Euclidean space expressed by:R^(c) ^(h) ^(r) ^(h) .As described above,R^(c) ^(h) ^(r) ^(h)is the orthogonal direct sum of R(W*) and N(W). FIG. 4 expresses R(W*)by one dimension, and two-dimensionally expresses N(W) as an orthogonalcomplementary space of R(W*) by two dimensions. The R(W*) component ofx_(p) is defined by the low-resolution block y and the reducedresolution matrix W. The N(W) component of x_(p) is obtained byorthogonally projecting z_(p) to N(W). All elements of a linear manifoldM parallel to N(W) satisfy the reconstruction constraint y=Wx. Thehigh-resolution block x_(p) obtained by this embodiment is generated byorthogonally projecting z_(p) to M, and is an element closest to z_(p)of those of M.

The generation data 162 is data required for this orthogonal projection.For example, the generation data 162 is data which represents W⁺ andI−W⁺W.

(1-3) Generation Method of Increased Resolution Block in IncreasedResolution Image 153 Generation Unit 105

The low-resolution block 155 is data which represents the aforementionedlow-resolution block y, and the high-resolution block 157 is data whichrepresents the aforementioned high-resolution block z_(p) at a positioncorresponding to the low-resolution block y. The generation data 162 isdata representing W⁺ described above and I−W⁺W. From these blocks, thegeneration unit 105 calculates an increased resolution block x_(p) inthe increased resolution image 153, which is indicated by the block data161, based on the aforementioned equation (x_(p)=W⁺y+(I−W⁺W)z_(p)).

(1-4) Effect of this Embodiment

This embodiment can reduce calculation cost since calculations are madefor each block in an image in place of the entire image. In the methodof the related art, for example, A. Tanaka, H. Imai, and M. Miyakoshi,“Digital Image Enlargement Based on Kernel Component Estimation,”International Journal of Computing Anticipatory Systems, vol. 15, pp.97-108, 2004, (w_(l)h_(l)+w_(h)h_(h)) product-sum calculations arerequired to calculate a pixel value of one pixel in X_(c) (i.e., tocalculate X_(c)=K+Y+(I−K⁺K)Z_(c)) upon calculating X_(c) from Y andZ_(c). On the other hand, in this embodiment, (c_(l)r_(l)+c_(h)r_(h))product-sum calculations are required per pixel due to:x _(p) =W ⁺ y+(I−W ⁺ W)z _(p).Since (c_(l)r_(l)+c_(h)r_(h))<(w_(l)h_(l)+w_(h)h_(h)), the calculationcost can be reduced. For example, when w_(l)=720, h_(l)=480, w_(h)=1920,and h_(h)=1080, since c_(l)=3, r_(l)=4, c_(h)=8, and r_(h)=9,c_(l)r_(l)+c_(h)r_(h)=84 and w_(l)h_(l)+w_(h)h_(h)=2419200, thus greatlyreducing the calculation volume.

This embodiment can reduce cost required to calculate a generalizedinverse matrix since calculations are made for each block in an image inplace of the entire image. In the method of the related art, forexample, A. Tanaka, H. Imai, and M. Miyakoshi, “Digital ImageEnlargement Based on Kernel Component Estimation,” International Journalof Computing Anticipatory Systems, vol. 15, pp. 97-108, 2004, a reducedresolution matrix K has a size as large as w_(l)h_(l) rows×w_(h)h_(h)columns. On the other hand, in this embodiment, the reduced resolutionmatrix W has a size of c_(l)r_(l) rows×c_(h)r_(h) columns. Thecalculation cost of the Moore-Penrose generalized inverse matrix can bereduced since it becomes higher with increasing matrix size, as isknown.

Furthermore, this embodiment can reduce the calculation cost since noiterative calculations by linear programming to uniquely define asolution which is not uniquely defined. In the method of the relatedart, for example, A. Tanaka, H. Imai, and M. Miyakoshi, “Digital ImageEnlargement Based on Kernel Component Estimation,” International Journalof Computing Anticipatory Systems, vol. 15, pp. 97-108, 2004, iterativecalculations by linear programming are required so as to maximize alikelihood based on the assumption that a differential image conforms tothe Laplace distribution. On the other hand, since this embodiment doesnot use any assumption that a differential image conforms to the Laplacedistribution, and does not require any iterative calculations by linearprogramming with high calculation cost, the calculation cost can bereduced accordingly.

In this embodiment, if the process of reduced resolution is given, andis expressed by an equation (y=Wx), the peak signal-to-noise ratio(PSNR) equal to or higher than z_(p) can be obtained. In order to provethis, let x be a high-resolution block before reduced resolution. Forthe sake of proof, it suffices to prove that SSD(x_(p)) of x_(p) withrespect to x becomes equal to or smaller than SSD(z_(p)) of z_(p) withrespect to x. SSD(x_(p)) can be calculated by:SSD(x _(p))=∥x−x _(p)∥²,SSD(x _(p))=W ⁺ y+(I−W ⁺ W)x−W ⁺ y−(I−W ⁺ W)z _(p)∥²,SSD(x _(p))=∥(I−W ⁺ W)(x−z _(p))∥²where ∥x∥ is a norm of x. SSD(z_(p)) can be calculated by:SSD(z _(p))=∥x−z _(p)∥²,SSD(z _(p))=∥W ⁺ y+(I−W ⁺ w)x−W ⁺ Wz _(p)−(I−W ⁺ W)z _(p)∥²,SSD(z _(p))=∥W ⁺ y−W ⁺ Wz _(p)∥²+2<W ⁺ y−W ⁺ Wz _(p),(I−W ⁺ W)(x−z_(p))>+∥(I−W ⁺ W)(x−z _(p))∥²SSD(z _(p))=∥W ⁺ y−W ⁺ Wz _(p)∥²∥+(I−W ⁺ W)(x−z _(p))∥²where <x, z> represents the inner product of x and z. The abovecalculation exploits the fact that<W ⁺ y−W ⁺ Wz _(p),(I−W ⁺ W)(x−z _(p))>=0holds because W⁺y−W⁺Wz_(p) is an element of R(W*), and (I−W⁺W)(x−z_(p))is an element of N(W), and R(W*) and N(W) are orthogonal to each other.Therefore, SSD(x_(p))−SSD(z_(p)) can be calculated by:SSD(x _(p))−SSD(z _(p))=∥(I−W ⁺ W)(x−z _(p))∥² −∥W ⁺ y−W ⁺ Wz_(p)∥²−∥(I−W ⁺ W)(x−z _(p))∥²,SSD(x _(p))−SSD(z _(p))=−∥W ⁺ y−W ⁺ Wz _(p)∥².Therefore, SSD(x_(p))−SSD(z_(p)) becomes equal to or smaller than zero.Hence, it can be proved that this embodiment can obtain the PSNR equalto or higher than z_(p) if the process of reduced resolution is given,and can be expressed by the equation (y=Wx).

(1-5) First Modification Modification of Generation Method ofHigh-Resolution Block

In this embodiment, the high-resolution block x_(p) which satisfies thereconstruction constraint is calculated by the equation(x_(p)=W⁺y+(I−W⁺W)z_(p)). By contrast, a high-resolution block whichstrictly does not satisfy the reconstruction constraint may becalculated. For example, the high-resolution block x_(p) may becalculated by:x _(p) =αW ⁺ y+(I−αW ⁺ W)z _(p).This equation expresses a straight line that passes through z_(p) andW⁺y+(I−W⁺W)z_(p). If α=0, the high-resolution block x_(p) equals z_(p).If α=1, the high-resolution block x_(p) is given by W⁺y+(I−W⁺W)z_(p). Ifα assumes a value between 0 and 1, the high-resolution block x_(p)becomes an interpolated point between z_(p) and W⁺y+(I−W⁺W)z_(p).Obtaining the interpolated point may often improve the subjective imagequality as compared with W⁺y+(I−W⁺W)z_(p). Because, W⁺y+(I−W⁺W)z_(p)often generates block noise like in resolution increasing using thenearest neighbor manner. However, by calculating an interpolated point,that block noise can be eliminated. Although SSD cannot improve,jagginess of edges can be reduced. In order to attain ×2 resolutionincreasing vertically and horizontally, letting W be:W=(¼ ¼ ¼ ¼).we have:W ⁺=(1 1 1 1)^(T).Hence, W⁺y is obtained by increasing in resolution y by the nearestneighbor manner. Note that T in this equation represents transpositionof a matrix. As can be seen from FIG. 4, since the resolution increasinggiven by (x_(p)=W⁺y+(I−W⁺W)z_(p)) is conversion that replaces the R(W*)component of z_(p) by W⁺y, it may generate block noise like inresolution increasing using the nearest neighbor manner. If aninterpolated point with z_(p) generated by another method capable ofgenerating a smooth image is calculated, that block noise can beeliminated.

Alternatively, in place of the high-resolution block that strictlysatisfies the reconstruction constraint, a high-resolution block thatminimizes:∥y−Wx∥²+β∥Dx∥²where D is a regularized matrix which has an arbitrary number of rows,and c_(h)r_(h) columns, and β is a regularization parameter may becalculated. This high-resolution block x_(p) can be calculated by:x _(p)=(W ^(T) W+βD ^(T) D)⁻¹ W ^(T) y+(I−W ⁺ W)z _(p)where −1 indicates an inverse matrix of the matrix. If β is positive,and D for which ∥Dx∥² is smaller as x is flatter is used, ahigh-resolution block flatter than that which strictly satisfies thereconstruction constraint is calculated. In this way, when an imageincludes noise, the noise can often be prevented from being emphasized.

Alternatively, by combining them, a high-resolution block may becalculated by:x _(p)=α(W ^(T) W+βD ^(T) D)⁻¹ W ^(T) y+(I−αW ⁺ W)z _(p).As a result, both block noise and noise emphasis can often beeliminated. Even when α and β, or α or β are or is added, thecalculation volume remains the same as that before addition.

(1-6) Second Modification Addition of Emphasis Unit 501

In the above embodiment, the arrangement is shown in FIG. 1, and thesequence of the processing is shown in FIG. 2. By contrast, in thismodification, the arrangement is changed to FIG. 5, and the sequence ofthe processing is changed to FIG. 6. Only changes will be explained. Inthis modification, step S601 is added to the sequence of the processing,and an emphasis unit 501 and low-resolution image input unit 502 areadded in place of the low-resolution image input unit 101 in thearrangement.

The low-resolution image input unit 502 inputs the same image as thelow-resolution image 151 in the above embodiment as an input, andnormally outputs the low-resolution image 151 to the emphasis unit 501as image data 551. When the low-resolution image 151 is encoded by,e.g., jpeg, mpeg2, or the like, the low-resolution image input unit 502decodes the low-resolution image 151, and outputs the image data 551 tothe emphasis unit 501.

The emphasis unit 501 receives the image data 551 from thelow-resolution image input unit 502, and applies emphasis processing tothis image data 551. Also, the emphasis unit 501 receives the block data158 from the setting unit 103, and passes a block 552 as an emphasizedblock, which is indicated by the block data 158, to the generation unit105. As the emphasis processing, for example, an unsharp mask orLaplacian filter is applied.

In an example of the operation shown in FIG. 6, only step S601 isdifferent from FIG. 2. In step S601, an emphasized image obtained byapplying the emphasis processing to the low-resolution image 151 isgenerated.

In step S204, the setting unit 103 sends the block data 158 to theemphasis unit 501 in this modification in place of the low-resolutionimage input unit 101. Also, the emphasis unit 501 sends a block in theemphasized image, which is indicated by the block data 158, to thegeneration unit 105 in this modification instead of the operation of theabove embodiment, in which the low-resolution image input unit 101 sendsthe low-resolution block 155 in the low-resolution image 151, which isindicated by the block data 158, to the generation unit 105.

In this modification, a low-resolution image is emphasized, and anincreased resolution image which becomes an emphasized low-resolutionimage if it is reduced in resolution is generated. In this way, asharper increased resolution image can be generated.

(1-7) Third Modification Modification Associated with Reduced ResolutionMatrix W

In the above embodiment, the initialization unit 104 calculates thereduced resolution matrix W. By contrast, in this modification, if theresolution increasing ratio is determined, the initialization unit 104may receive W from outside the image resolution increasing apparatus100. Only changes in such case will be explained. In this case, forexample, a wire connection for inputting W may be added to theinitialization unit 104 in FIG. 1. A block diagram thereof will beomitted. As the sequence of the processing, upon calculation of thegeneration data 162 by the initialization unit 104 in step S201, input Wis used in place of calculating W. In this manner, the calculation costof W can be reduced.

(1-8) Fourth Modification Another Modification Associated with ReducedResolution Matrix W

In the above embodiment, the initialization unit 104 calculates thereduced resolution matrix W. By contrast, W may be stored in theinitialization unit 104. For example, W is calculated in advance, andthe initialization unit 104 stores W. Only changes in this case will beexplained. In this case, as the sequence of the processing, uponcalculation of the generation data 162 by the initialization unit 104 instep S201, W stored in advance is used in place of calculating W. Inthis manner, the calculation cost upon calculating W can be reduced. Thegeneration data 162 may be stored in place of the reduced resolutionmatrix W. In this case, the initialization unit 104 may store thegeneration data 162 or the generation unit 105 may store it. When thegeneration unit 105 stores the generation data 162, the calculation costupon calculating the generation data 162 by the initialization unit 104can be reduced. Also, the need for sending the generation data 162 tothe generation unit 105 can be obviated.

(1-9) Fifth Modification Modification Under Assumption of AnotherProcess of Reduced Resolution

In this embodiment, the process of reduced resolution is given by:y=Wx.By contrast, in consideration of additive noise, the process of reducedresolution may be expressed by:y=Wx+nwhere n is a (c_(l)r_(l))-dimensional column vector which represents ablock of additive noise. Let Q be a correlation matrix of {n}. That is,Q is defined by:Q=E _(n)(n

n )where E_(n) is an average associated with n. Since n is a real numbervector, we have:Q=E _(n)(nn ^(T))where n^(T) is a transposed vector of n. In case of consideration ofadditive noise, a high-resolution block x_(p) is calculated by:x _(p) =V ⁺ W*U ⁺ y+(I−W ⁺ W)z _(p)where U and V are matrices, which are respectively defined by:U=WW*+Q,V=W*U ⁺ W.The first term (R(W*) component) of an equation(x_(p)=V⁺W*U⁺y+(I−W⁺W)z_(p)) is obtained by applying a projection filterdescribed in “Hidemitsu Ogawa, “Reconstruction of Signal and Image[III]—Projection Filters for Optimal Reconstruction—,” Journal of IEICE,vol. 71, pp. 739-748, 1988” to resolution increasing of an image forrespective blocks. Since the projection filter minimizes the average ofnoise in R(W*), the first term (R(W*) component) of the equation(x_(p)=V⁺W*U⁺y+(I−W⁺W)z_(p)) is the best approximation in R(W*). Bymerely applying the projection filter to resolution increasing of animage for respective blocks, only the R(W*) component is estimated. Bycontrast, in this modification, since the N(W) component is alsoestimated by the second term of the equation(x_(p)=V⁺W*U⁺y+(I−W⁺W)z_(p)), a sharper image can be obtained than acase in which the projection filter is applied to resolution increasingof an image for respective blocks. To attain this change, theinitialization unit 104 need only replace the generation data 162 bydata that expresses V⁺W*U⁺ and I−W⁺W above. Note that when R(W) matchesR^(c) ^(l) ^(r) ^(l)the projection filter matches a generalized inverse filter. When thereis no additive noise, Q becomes a matrix having only zero as elements,and the projection filter matches a generalized inverse filter.Conversely, the fifth modification is effective when R(W) does not matchR^(c) ^(l) ^(r) ^(l) .

(1-10) Sixth Modification Modification Under Assumption of Still AnotherProcess of Reduced Resolution

The above embodiment adopts, as the reconstruction constraint, theequation (y=Wx) that expresses the process of reduced resolution of ac_(h)×r_(h) high-resolution block x to a c_(l)×r_(l) low-resolutionblock. By contrast, an equation that expresses the process of reducedresolution of a (c_(h)+b)×(r_(h)+b) high resolution block x to ac_(l)×r_(l) low-resolution block y may be adopted as the reconstructionconstraint. The process of reduced resolution (reconstructionconstraint) in this case is given by:y=ABxwhere a high-resolution block x is a ((c_(h)+b)(r_(h)+b))-dimensionalcolumn vector, B is a matrix of c_(h)r_(h) rows×(c_(h)+b)(r_(h)+b)columns, and A is a matrix of c_(l)r_(l) rows×c_(h)r_(h) columns. Thematrix B represents a state in which the high-resolution block x blurs,and the matrix A represents a state in which a high-resolution block Bxis reduced in resolution by an area average matrix. FIG. 7 shows anexample of the process of reduced resolution (reconstruction constraint)in this case. In this example, b=2, c_(h)=4, r_(h)=4, c_(l)=1, andr_(l)=1.If we have:W=AB,the process of reduced resolution (reconstruction constraint) is givenby:y=Wx.Hence, the problem can be handled using the same equation as that whenthis change is not applied. Therefore, to attain this change, theinitialization unit 104 need only substitute the generation data 162 bydata which represents W⁺ based on new W and I−W⁺W. Note that the blocksizes of the high-resolution block x and low-resolution block y aredifferent, and the reduced resolution matrix W has a different size. Adetailed description will be given taking a case of ×2 resolutionincreasing vertically and horizontally as an example. When thismodification is not applied, for example, neighboring high-resolutionblocks 901 and 902 of a high-resolution image shown in FIG. 9 aregenerated from neighboring low-resolution blocks 801 and 802 of alow-resolution image shown in FIG. 8, respectively. On the other hand,when this modification is applied to have b=2, overlappinghigh-resolution blocks 1001 and 1002 of a high-resolution image shown inFIG. 10 are generated from the neighboring low-resolution blocks 801 and802 of the low-resolution image shown in FIG. 8, respectively.Therefore, to attain this change, the setting unit 103 needs to changethe block data 159 and 161 to be controlled. The high-resolution block157 changes accordingly. The generation unit 105 may be changed to adoptaverage values for the overlapping part of the high-resolution blocks1001 and 1002. Upon further considering additive noise, the modificationof (1-8) may be applied.

(1-11) Use Example of this Embodiment

Upon implementing resolution increasing vertically and horizontally,respectively, the calculation cost can be reduced more when theresolution increasing is applied vertically only, and is then appliedhorizontally only. This is because the size of the reduced resolutionmatrix W used when the resolution increasing is applied independentlyvertically and horizontally is smaller than that used when theresolution increasing is applied simultaneously vertically andhorizontally, and the cost for the calculation of W⁺ and that of(x_(p)=W⁺y+(I−W⁺W)z_(p)) of the independent application is smaller.

SECOND EMBODIMENT

An image resolution increasing apparatus of this embodiment will bedescribed below with reference to FIG. 11.

An image resolution increasing apparatus 1100 of this embodimentincludes an initialization unit 1101, interpolation coefficientinitialization unit 1102, coefficient calculation unit 1103, settingunit 1104, low-resolution image input unit 1105, and generation unit1106. The image resolution increasing apparatus 1100 generates andoutputs an increased resolution image 1155 based on size data 160 and alow-resolution image 151.

The initialization unit 1101 acquires size data 160, and outputs blocksize data 154.

The interpolation coefficient initialization unit 1102 acquires theblock size data 154, initializes that block size data 154, andcalculates interpolation coefficient data 1151. The interpolationcoefficient data 1151 is data which represents coefficients used uponcalculating a c_(h)×r_(h) high-resolution block z_(p) from a(c_(l)+i)×(r_(l)+i) low-resolution block y_(p) by an interpolationblock, and is denoted by reference symbol C, as will be described later.i is a value indicating how many surrounding pixels are to be used inthat interpolation manner so as to interpolate a pixel. For example,when the interpolation manner is a cubic convolution manner, i=4, andwhen it is a bilinear manner, i=2. It is called an interpolationcoefficient matrix and is represented by C.z_(p)=Cy_(p)where the high-resolution block z_(p) is a (c_(h)r_(h))-dimensionalcolumn vector, a low-resolution block y_(p) is a((c_(l)+i)(r_(l)+i))-dimensional column vector, and the interpolationcoefficient matrix C is a matrix of c_(h)r_(h) rows×(c_(l)+i)(r_(l)+i)columns. FIG. 13 is a view for explaining the relationship between y_(p)and z_(p). As can be seen from FIG. 13, y_(p) represents a part broaderthan that of an object represented by z_(p). y_(p) is a low-resolutionblock required to generate z_(p) by the interpolation manner. In thefirst embodiment, since the high-resolution image 152 is input, thehigh-resolution block 157 (z_(p)) can be obtained. On the other hand, inthis embodiment, no high-resolution image 152 is input. This embodimentcalculates x_(p) obtained under the assumption that z_(p) correspondingto the high-resolution block 157 in the first embodiment is generatedbased on y_(p) and the interpolation manner. Upon calculation of thisx_(p), x_(p) is directly calculated without the intervention of z_(p).The reason why the calculation of z is not intervened is to reduce thecalculation volume. The interpolation coefficient matrix C is requiredto directly calculate x_(p).

The coefficient calculation unit 1103 acquires the interpolationcoefficient data 1151 and block size data 154, and calculatescoefficient data 1152. The coefficient data 1152 is required to generatethe c_(h)×r_(h) increased resolution block x_(p) from the(c_(l)+i)×(r_(l)+i) low-resolution block y_(p), and is a coefficientmatrix T (described later) calculated based on W and C above. T iscalculated, for example, by:T=S+(I−W ⁺ W)Cwhere S is a matrix which satisfies Sy_(p)=W⁺y and has as its elements,the elements of W⁺ and zero. For example, when both c_(l) and r_(l) are“1”, S=[0 W⁺ 0]. [0 W⁺ 0] is a matrix obtained by coupling zero matriceswith the same size to the right and left sides of W⁺.

The setting unit 1104 acquires image number data 156 and the block sizedata 154, and sets block data 1153 and 161. The block data 1153 is dataindicating the position of a low-resolution block 1154 having a widthc_(l)+i and height r_(l)+i in the low-resolution image 151. The blockdata 161 is data indicating the position of an increased resolutionblock having a width c_(h) and height r_(h) in the increased resolutionimage 1155. The block data 161 is designated at a position correspondingto the block data 1153.

The low-resolution image input unit 1105 acquires the low-resolutionimage 151 and block data 1153, and generates the low-resolution block1154 in the low-resolution image 151, which is indicated by the blockdata 1153. The block data 1153 is designated while shifting itsposition. More specifically, the position is shifted by c_(l)horizontally and r_(l) vertically. In this way, the low-resolutionblocks 1154 in the low-resolution image 151 are processed in turn. If iis positive, the low-resolution blocks 1154 overlap each other. When alow-resolution image to be increased in resolution (moving pictureframes or still pictures) is encoded by, e.g., jpeg, mpeg2, or the like,the low-resolution image input unit 1105 comprises a codec for decodingthe encoded image.

The generation unit 1106 generates an increased resolution block in theincreased resolution image 1155, which is indicated by the block data161, based on the low-resolution block 1154 in the low-resolution image151 and the coefficient data 1152. The increased resolution block x_(p)in the increased resolution image 1155 is generated based on thelow-resolution block y_(p) and the coefficient matrix T by:x_(p)=Ty_(p).This x_(p) satisfies:x _(p) =W ⁺ y+(I−W ⁺ W)Cy _(p).Therefore, if an equation (z_(p)=Cy_(p)) holds for the high-resolutionblock z_(p) of the high-resolution image input in the first embodiment,the second embodiment can obtain the same increased resolution block xas in the first embodiment. Even when the equation (z_(p)=Cy_(p)) doesnot hold for the high-resolution block z_(p) of the high-resolutionimage, the proof in (1-4) does not assume anything about z_(p). Hence,if the process of reduced resolution is given by the equation (y=Wx),SSD(x_(p)) of x_(p) with respect to x becomes equal to or smaller thanSSD(Cy_(p)) of the high-resolution block Cy_(p) obtained by theinterpolation manner with respect to x. FIG. 14 is a view for explainingthe relationship between y_(p) and x_(p). y_(p) obviously represent aportion wider than the object portion represented by x_(p). y_(p) is alow-resolution block required to generate x_(p) using the coefficientmatrix T of this embodiment. x_(p) is directly calculated from y_(p) andT without any intervention of the calculation of z_(p).

An example of the operation of the image resolution increasing apparatus1100 shown in FIG. 11 will be described below with reference to FIG. 12.

(S1201) The image number data 156 is input to the setting unit 1104, andthe size data 160 is input to the initialization unit 1101. Theinitialization unit 1101 calculates the block size data 154, and sendsit to the interpolation coefficient initialization unit 1102,coefficient calculation unit 1103, and setting unit 1104.

(S1202) The interpolation coefficient initialization unit 1102initializes the block size data 154 to generate the interpolationcoefficient data 1151, and sends it to the coefficient calculation unit1103.

(S1203) The coefficient calculation unit 1103 calculates the coefficientdata 1152 based on the interpolation coefficient data 1151 and blocksize data 154, and sends it to the generation unit 1106.

(S1204) The low-resolution image input unit 1105 receives the n-th image(n=1, 2, . . . , N) of low-resolution images to be increased inresolution (moving picture frames or still pictures) as thelow-resolution image 151.

(S1205) The setting unit 1104 sends the block data 1153 to thelow-resolution image input unit 1105, and the block data 161 to thegeneration unit 1106. The low-resolution image input unit 1105 sends thelow-resolution block 1154 in the low-resolution image 151, which isindicated by the block data 1153, to the generation unit 1106. The blockdata 1153 is designated while shifting its position upon repetitivelyprocessing step S1205. More specifically, the position is shifted byc_(l) horizontally and r_(l) vertically. In this way, the low-resolutionblocks 1154 in the low-resolution image 151 are processed in turn. If iis positive, the low-resolution blocks 1154 overlap each other. Theposition of the block data 161 is shifted by c_(h) horizontally andr_(h) vertically in practice. By associating the block data 1153 and 161with each other, the processing is done for each image block in place ofthe entire image.

(S1206) The generation unit 1106 generates an increased resolution blockin the increased resolution image 1155, which is indicated by the blockdata 161, based on the high-resolution block 1154 and coefficient data1152.

(S1207) The setting unit 1104 checks if all increased resolution blocksin the increased resolution image 1155 have been processed. If all theblocks have been processed, the process advances to step S1208;otherwise, the process returns to step S1205. Step S1206 is repeatedwhile the setting unit 1104 changes the block data 1153 and 161 to becontrolled. As a result of this repetition, the generation unit 1106generates and outputs the increased resolution image 1155 by generatingall increased resolution blocks in the increased resolution image 1155.Note that end blocks are hard to receive attention, and are often notimportant. Hence, these blocks may be generated by a method requiringlower calculation cost.

(S1208) The setting unit 1104 checks based on the image number data 156if all (N) low-resolution images to be increased in resolution (movingpicture frames or still pictures) have been processed. If all the imageshave been processed, the processing ends; otherwise, the process returnsto step S1204.

(2-1) Effect of this Embodiment

This embodiment requires the calculation cost fewer than the firstembodiment. The first embodiment requires (c_(l)r_(l)+c_(h)r_(h))product-sum calculations per pixel. This embodiment requires(c_(l)+i)(r_(l)+i) product-sum calculations from the equation(x_(p)=Ty_(p)). In most cases, since(c_(l)+i)(r_(l)+i)<(c_(l)r_(l)+c_(h)r_(h)), the calculation cost can bereduced. For example, when w_(l)=720, h_(l)=480, w_(h)=1920, h_(h)=1080,and i=4, since c_(l)=3, r_(l)=4, c_(h)=8, and r_(h)=9,(c_(l)+i)(r_(l)+i)=56, and c_(l)r_(l)+c_(h)r_(h)=84. Hence, thecalculation volume can be further reduced compared to the firstembodiment. When resolution increasing is applied not simultaneouslyvertically and horizontally but independently in these directions, thecalculation cost can be further reduced. FIG. 15 shows an example of astate in which the resolution increasing is independently appliedvertically and horizontally. In this example, w_(l)=720, h_(l)=480,w_(h)=1920, h_(h)=1080, and i=4. In the example of FIG. 15, two pixelseach protrude from 3×4 and 3×9 blocks. The protrusion amount isdetermined by i specified depending on the interpolation manner. Theexample of FIG. 15 is the case of the interpolation manner of i=4(bicubic manner, cubic convolution manner, or the like). For example,when a bilinear manner of i=2 is used as the interpolation manner, onepixel each protrudes from 3×4 and 3×9 blocks.

(2-2) First Modification Modification of Generation Method ofHigh-Resolution Block

As in the first modification of the first embodiment in (1-5), ahigh-resolution block that does not satisfy the reconstructionconstraint may be calculated. For example, T may be given by:T=αS+(I−αW ⁺ W)C.Alternatively, by exploiting S that satisfiesSy_(p)=(W^(T)W+βD^(T)D)⁻¹W^(T)y, T may be given by:T=S+(I−W ⁺ W)C.Alternatively, by exploiting S that satisfiesSy_(p)=(W^(T)W+βD^(T)D)⁻¹W^(T)y, T may be given by:T=αS+(I−αW ⁺ W)C.As a result, the same effect as in (1-5) can be obtained.

(2-3) Second Modification Modification Added with Emphasis Unit

In this embodiment, the processing sequence is FIG. 12, and thearrangement is FIG. 11. By contrast, the arrangement may be changed toFIG. 16, and the processing sequence may be changed to FIG. 17. Onlychanges will be described. In this modification, step S1401 is added tothe processing sequence, and an emphasis unit 1601 is added to thearrangement.

The emphasis unit 1601 acquires the low-resolution image 151 as imagedata 551, and acquires the block data 1153 from the setting unit 1104.The emphasis unit 1601 extracts a low-resolution block 155 indicated bythe block data 1153, applies emphasis processing to this low-resolutionblock 155, and passes an emphasized block 1651 to the generation unit1106. As the emphasis processing, for example, an unsharp mask orLaplacian filter is applied.

An example of the operation of an image resolution increasing apparatus1600 shown in FIG. 16 will be described below with reference to FIG. 17.

(S1204) In addition to the aforementioned processing, a low-resolutionimage input unit 502 outputs image data 551, and sends it to theemphasis unit 1601. The image data 551 is obtained by arranging pixelvalues of the low-resolution image 151. Therefore, if the low-resolutionimage 151 is not encoded, the image data 551 is the low-resolution image151 itself; otherwise, the image data 551 is obtained by decoding thelow-resolution image 151.

(S1401) An emphasis image is generated by applying the emphasisprocessing to the image data 551. As the emphasis processing, forexample, an unsharp mask or Laplacian filter is applied.

(S1205) The setting unit 1103 sends the block data 1153 of the secondembodiment to the emphasis unit 1601 in this modification in place ofthe low-resolution image input unit 1105. The emphasis unit 1601 sendsthe low-resolution block 1651 in the emphasized image, which isindicated by the block data 1153, to the generation unit 1106 in placeof the operation of the second embodiment in which the low-resolutionimage input unit 1105 sends the low-resolution block 1154 in thelow-resolution image 151, which is indicated by the block data 1153, tothe generation unit 1106.

In this modification, a low-resolution image is emphasized, and anincreased resolution image which becomes an emphasized low-resolutionimage if it is reduced in resolution is generated. In this way, asharper increased resolution image can be generated.

(2-4) Third Modification Modification Associated with CoefficientCalculation Unit 1103

The coefficient calculation unit 1103 may be changed to calculate thecoefficient matrix T (coefficient data 1152) by exploiting S thatsatisfies Sy_(p)=W⁺Ey. Note that j is, for example, 2, and E is a matrixof c_(l)r_(l) rows×(c_(l)+j)(r_(l)+j) columns. The matrix E is anemphasis matrix which converts a (c_(l)+j)×(r_(l)+j) low-resolutionblock into a low-resolution block in which central c_(l)×r_(l) pixelsare emphasized by applying a 3×3 emphasis filter (unsharp mask orLaplacian filter). With this change, without adding any emphasis unitunlike in the second modification of (2-3), an increased resolutionimage which becomes an emphasized low-resolution image by reducedresolution can be generated. With this change, a sharper increasedresolution image can be generated.

(2-5) Fourth Modification Modification Under Assumption of AnotherProcess of Reduced Resolution

In the second embodiment, the process of reduced resolution is given by:y=Wx.By contrast, as in the fifth modification of the first embodiment in(1-9), additive noise may be considered, and the process of reducedresolution may be expressed by:y=Wx+n.In this case, it may be changed to calculate the coefficient matrix T(coefficient data 1152) by exploiting S that satisfies Sy_(p)=V⁺W*U⁺y.In this way, the same effect as in (1-9) can be obtained.

(2-6) Fifth Modification Modification Under Assumption of Still AnotherProcess of Reduced Resolution

As in the sixth modification of the first embodiment in (1-10), theprocess of reduced resolution (reconstruction constraint) may also bedescribed by:y=ABxwhere a high-resolution block x is a ((c_(h)+b)(r_(h)+b))-dimensionalcolumn vector, B is a matrix of c_(h)r_(h) rows×(c_(h)+b)(r_(h)+b)columns, and A is a matrix of c_(l)r_(l) rows×c_(h)r_(h) columns. If wehave:W=AB,the process of reduced resolution (reconstruction constraint) can beexpressed by:y=Wx.Hence, the problem can be handled using the same equation as that whenthis change is not applied. Therefore, in the coefficient matrix T(coefficient data 1152), a part associated with W in the equation(T=S+(I−W+W)C)) may be substituted. However, as in (1-10), note that theblock sizes of the high-resolution block x and low-resolution block yare different, and the reduced resolution matrix W has a different size.

(2-7) Sixth Modification Modification of Calculation of Coefficient Data1152 in Advance

If a resolution increasing ratio is determined in advance, thecoefficient data 1152 may be calculated in advance. In this case, theimage resolution increasing apparatus 1100 shown in FIG. 11 may bedivided into two apparatuses: a coefficient calculation apparatus 1800shown in FIG. 18 and a resolution increasing calculation apparatus 1900shown in FIG. 19. For example, when the image resolution increasingapparatus 1100 of this embodiment is applied to a television, one offacilities required to manufacture televisions in a factory is thecoefficient calculation apparatus 1800, and one of components equippedin each television is the resolution increasing calculation apparatus1900. In this case, steps S1201 to S1203 in FIG. 12 are the processes inthe coefficient calculation apparatus 1800, and steps S1204 to S1208 arethe processes in the resolution increasing calculation apparatus 1900.As a result, since the resolution increasing calculation apparatus 1900need not calculate the coefficient data 1152, the calculation cost canbe reduced.

According to the aforementioned embodiment, since the processing is donefor respective blocks and the numbers of multiplications and additionsrequired to calculate the pixel value of one pixel in a high-resolutionimage are reduced, the calculation cost can be reduced. Since the sizeof the reduced resolution matrix is small, the cost required tocalculate a Moore-Penrose generalized inverse matrix of the reducedresolution matrix is reduced. Furthermore, any likelihood based on theassumption that a differential image conforms to the Laplacedistribution need not be maximized.

The image resolution increasing apparatus of this embodiment can be usedto provide a function of increasing the image quality to electronicapparatuses having functions of recording, displaying, and printingimages such as a personal computer, digital camera, video camera,portable phone, portable information terminal, printer, and the like.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. An image resolution increasing method, comprising: inputting alow-resolution image having a first size as a size that indicates thenumbers of pixels arranged vertically and horizontally; inputting ahigh-resolution image obtained by increasing in resolution thelow-resolution image to a second size by an interpolation manner or asuper-resolution manner, the second size being a desired size of a firstimage finally generated; calculating a first block size corresponding tothe first size and a second block size corresponding to the second sizein accordance with the first size and the second size; setting a firstposition of a block of the first block size in the low-resolution image,and a second position of a block of the second block size in thehigh-resolution image or the first image, the second positioncorresponding to the first position; setting a first block which isincluded in the low-resolution image and is located at the firstposition, and a second block which is included in the high-resolutionimage and is located at the second position; setting, as an increasedresolution block of the first block, a third block expressed by a secondvector obtained by projecting a first vector representing the secondblock to a linear manifold as a set of vectors that indicate fourthblocks of the second block size, the fourth blocks becoming the firstblock due to reduced resolution, in a Euclidean space having, as thenumber of dimensions, a product of the number of pixels arrangedvertically in the second block size and the number of pixels arrangedhorizontally in the second block size to obtain a plurality of increasedresolution blocks; and generating the first image using at least one ofthe increased resolution blocks set by changing the first position andthe second position.
 2. The method according to claim 1, whereincalculating the first block size calculates the first block size and thesecond block size based on a resolution increasing ratio from thelow-resolution image to the first image.
 3. The method according toclaim 1, further comprising emphasizing the low-resolution image.
 4. Animage resolution increasing method, comprising: inputting alow-resolution image having a first size as a size that indicates thenumbers of pixels arranged vertically and horizontally; inputting ahigh-resolution image obtained by increasing in resolution thelow-resolution image to a second size by an interpolation manner or asuper-resolution manner, the second size being a desired size of a firstimage finally generated; calculating a first block size corresponding tothe first size and a second block size corresponding to the second sizein accordance with the first size and the second size; setting a firstposition of a block of the first block size in the low-resolution image,and a second position corresponding to the first position of the secondblock size in the high-resolution image or the first image; setting afirst block which is included in the low-resolution image and is locatedat the first position, and a second block which is included in thehigh-resolution image and is located at the second position; setting, asan increased resolution block of the first block, a third blockexpressed by a vector corresponding to points on a straight line thatpasses through two points indicated by a first vector and a secondvector, the first vector representing the second block, the secondvector being obtained by projecting the first vector to a linearmanifold as a set of vectors that indicate fourth blocks of the secondblock size, the fourth blocks becoming the first block due to reducedresolution, in a Euclidean space having, as the number of dimensions, aproduct of the number of pixels arranged vertically in the second blocksize and the number of pixels arranged horizontally in the second blocksize to obtain a plurality of increased resolution blocks; andgenerating the first image using at least one of the increasedresolution blocks set by changing the first position and the secondposition.
 5. The method according to claim 1, wherein when a pluralityof increased resolution blocks based on the third block overlap eachother, average values of pixel values between overlapping blocks areadopted as pixel values of the overlapping parts.
 6. An image resolutionincreasing method, comprising: calculating a first block sizecorresponding to a first size and a second block size corresponding to asecond size in accordance with the first size and the second size, thefirst size being a size of a low-resolution image and a size thatindicates the numbers of pixels arranged vertically and horizontally,the second size being a desired size of a first image finally generated;setting a first position of a block of the first block size in thelow-resolution image, and a second position of a block of the secondblock size in the first image, the second position corresponding to thefirst position; setting a first block which is included in thelow-resolution image and is located at the first position; calculating,using the first block size and the second block size, a firstcoefficient used to generate a second block of the second block size byan interpolation manner from the first block; calculating, using thefirst coefficient, the first block size, and the second block size, asecond coefficient used to calculate a second vector obtained byprojecting a first vector indicating the second block to a linearmanifold as a set of vectors that indicate third blocks of the secondblock size, the third blocks becoming a part of the first block due toreduced resolution, in a Euclidean space having, as the number ofdimensions, a product of the number of pixels arranged vertically in thesecond block size and the number of pixels arranged horizontally in thesecond block size; calculating the second vector corresponding to aposition indicated by the second position from the first block and thesecond coefficient, setting a fourth block indicated by the secondvector to an increased resolution block as a part of the first block toobtain a plurality of increased resolution blocks; and generating thefirst image using at least one of the increased resolution blocks set bychanging the first position and the second position.
 7. The methodaccording to claim 6, wherein calculating the first block sizecalculates the first block size and the second block size based on aresolution increasing ratio from the low-resolution image to the firstimage.
 8. The method according to claim 6, further comprisingemphasizing the low-resolution image.
 9. An image resolution increasingmethod, comprising: calculating a first block size corresponding to afirst size and a second block size corresponding to a second size inaccordance with the first size and the second size, the first size beinga size of a low-resolution image and a size that indicates the numbersof pixels arranged vertically and horizontally, the second size being adesired size of a first image finally generated; setting a firstposition of a block of the first block size in the low-resolution image,and a second position of a block of the second block size in the firstimage, the second position corresponding to the first position; settinga first block which is included in the low-resolution image and islocated at the first position; calculating, using the first block sizeand the second block size, a first coefficient used to generate a secondblock of the second block size by an interpolation manner from the firstblock; calculating, using the first coefficient, the first block size,and the second block size, a second coefficient used to calculate asecond vector obtained by projecting a first vector indicating thesecond block to a linear manifold as a set of vectors that indicatethird blocks of the second block size, the third blocks becoming a partof the first block due to reduced resolution, in a Euclidean spacehaving, as the number of dimensions, a product of the number of pixelsarranged vertically in the second block size and the number of pixelsarranged horizontally in the second block size; calculating the secondvector corresponding to a position indicated by the second position fromthe first block and the second coefficient; setting, as an increasedresolution block as a part of the first block, a fourth block expressedby a vector corresponding to points on a straight line that passesthrough two points indicated by the first vector and the second vectorto obtain a plurality of increased resolution blocks; and generating thefirst image using at least one of the increased resolution blocks set bychanging the first position and the second position.
 10. The methodaccording to claim 6, wherein when a plurality of increased resolutionblocks based on the fourth blocks overlap each other, average values ofpixel values between overlapping blocks are adopted as pixel values ofthe overlapping parts.
 11. An image resolution increasing apparatus,comprising: a first input unit configured to input a low-resolutionimage having a first size as a size that indicates the numbers of pixelsarranged vertically and horizontally; a second input unit configured toinput a high-resolution image obtained by increasing in resolution thelow-resolution image to a second size by an interpolation manner or asuper-resolution manner, the second size being a desired size of a firstimage finally generated; a calculation unit configured to calculate afirst block size corresponding to the first size and a second block sizecorresponding to the second size in accordance with the first size andthe second size; a first setting unit configured to set a first positionof a block of the first block size in the low-resolution image, and asecond position of a block of the second block size in thehigh-resolution image or the first image, the second positioncorresponding to the first position; a second setting unit configured toset a first block which is included in the low-resolution image and islocated at the first position, and a second block which is included inthe high-resolution image and is located at the second position; a thirdsetting unit configured to set, as an increased resolution block of thefirst block, a third block expressed by a second vector obtained byprojecting a first vector representing the second block to a linearmanifold as a set of vectors that indicate fourth blocks of the secondblock size, the fourth blocks becoming the first block due to reducedresolution, in a Euclidean space having, as the number of dimensions, aproduct of the number of pixels arranged vertically in the second blocksize and the number of pixels arranged horizontally in the second blocksize to obtain a plurality of increased resolution blocks; and ageneration unit configured to generate the first image using at leastone of the increased resolution blocks set by changing the firstposition and the second position.
 12. The apparatus according to claim11, wherein the calculation unit configured to calculate the first blocksize and the second block size based on a resolution increasing ratiofrom the low-resolution image to the first image.
 13. The apparatusaccording to claim 11, further comprising an emphasis unit configured toemphasize the low-resolution image.
 14. An image resolution increasingapparatus, comprising: a first input unit configured to input alow-resolution image having a first size as a size that indicates thenumbers of pixels arranged vertically and horizontally; a second inputunit configured to input a high-resolution image obtained by increasingin resolution the low-resolution image to a second size by aninterpolation manner or a super-resolution manner, the second size beinga desired size of a first image finally generated; a calculation unitconfigured to calculate a first block size corresponding to the firstsize and a second block size corresponding to the second size inaccordance with the first size and the second size; a first setting unitconfigured to set a first position of a block of the first block size inthe low-resolution image, and a second position corresponding to thefirst position of the second block size in the high-resolution image orthe first image; a second setting unit configured to set a first blockwhich is included in the low-resolution image and is located at thefirst position, and a second block which is included in thehigh-resolution image and is located at the second position; a thirdsetting unit configured to set, as an increased resolution block of thefirst block, a third block expressed by a vector corresponding to pointson a straight line that passes through two points indicated by a firstvector and a second vector, the first vector representing the secondblock, the second vector being obtained by projecting the first vectorto a linear manifold as a set of vectors that indicate fourth blocks ofthe second block size, the fourth blocks becoming the first block due toreduced resolution, in a Euclidean space having, as the number ofdimensions, a product of the number of pixels arranged vertically in thesecond block size and the number of pixels arranged horizontally in thesecond block size to obtain a plurality of increased resolution blocks;and a generation unit configured to generate the first image using atleast one of the increased resolution blocks set by changing the firstposition and the second position.
 15. The apparatus according to claim11, wherein when a plurality of increased resolution blocks based on thethird block overlap each other, average values of pixel values betweenoverlapping blocks are adopted as pixel values of the overlapping parts.16. An image resolution increasing apparatus, comprising: a firstcalculation unit configured to calculate a first block sizecorresponding to a first size and a second block size corresponding to asecond size in accordance with the first size and the second size, thefirst size being a size of a low-resolution image and a size thatindicates the numbers of pixels arranged vertically and horizontally,the second size being a desired size of a first image finally generated;a first setting unit configured to set a first position of a block ofthe first block size in the low-resolution image, and a second positionof a block of the second block size in the first image, the secondposition corresponding to the first position; a second setting unitconfigured to set a first block which is included in the low-resolutionimage and is located at the first position; a second calculation unitconfigured to calculate, using the first block size and the second blocksize, a first coefficient used to generate a second block of the secondblock size by an interpolation manner from the first block; a thirdcalculation unit configured to calculate, using the first coefficient,the first block size, and the second block size, a second coefficientused to calculate a second vector obtained by projecting a first vectorindicating the second block to a linear manifold as a set of vectorsthat indicate third blocks of the second block size, the third blocksbecoming a part of the first block due to reduced resolution, in aEuclidean space having, as the number of dimensions, a product of thenumber of pixels arranged vertically in the second block size and thenumber of pixels arranged horizontally in the second block size; a thirdsetting unit configured to calculate the second vector corresponding toa position indicated by the second position from the first block and thesecond coefficient, and to set a fourth block indicated by the secondvector to an increased resolution block as a part of the first block toobtain a plurality of increased resolution blocks; and a generation unitconfigured to generate the first image using at least one of theincreased resolution blocks set by changing the first position and thesecond position.
 17. The apparatus according to claim 16, wherein thefirst calculation unit configured to calculate the first block size andthe second block size based on a resolution increasing ratio from thelow-resolution image to the first image.
 18. The apparatus according toclaim 16, further comprising an emphasis unit configured to emphasizethe low-resolution image.
 19. An image resolution increasing apparatus,comprising: a first calculation unit configured to calculate a firstblock size corresponding to a first size and a second block sizecorresponding to a second size in accordance with the first size and thesecond size, the first size being a size of a low-resolution image and asize that indicates the numbers of pixels arranged vertically andhorizontally, the second size being a desired size of a first imagefinally generated; a first setting unit configured to set a firstposition of a block of the first block size in the low-resolution image,and a second position of a block of the second block size in the firstimage, the second position corresponding to the first position; a secondsetting unit configured to set a first block which is included in thelow-resolution image and is located at the first position; a secondcalculation unit configured to calculate, using the first block size andthe second block size, a first coefficient used to generate a secondblock of the second block size by an interpolation manner from the firstblock; a third calculation unit configured to calculate, using the firstcoefficient, the first block size, and the second block size, a secondcoefficient used to calculate a second vector obtained by projecting afirst vector indicating the second block to a linear manifold as a setof vectors that indicate third blocks of the second block size, thethird blocks becoming a part of the first block due to reducedresolution, in a Euclidean space having, as the number of dimensions, aproduct of the number of pixels arranged vertically in the second blocksize and the number of pixels arranged horizontally in the second blocksize; a fourth calculation unit configured to calculate the secondvector corresponding to a position indicated by the second position fromthe first block and the second coefficient; a third setting unitconfigured to set, as an increased resolution block as a part of thefirst block, a fourth block expressed by a vector corresponding topoints on a straight line that passes through two points indicated bythe first vector and the second vector to obtain a plurality ofincreased resolution blocks; and a generation unit generating the firstimage using at least one of the increased resolution blocks set bychanging the first position and the second position.
 20. The apparatusaccording to claim 16, wherein when a plurality of increased resolutionblocks based on the fourth blocks overlap each other, average values ofpixel values between overlapping blocks are adopted as pixel values ofthe overlapping parts.